소프트웨어 테스팅의 교훈들

293개의 교훈으로 이루어진 책.

Any one of these tiny lessons could be worth the price of the book —Tom DeMarco

Ward Cunningham의 Framework for integrated test도 이 책에서 말하는 Context-driven testing의 영향을 많이 받았다고 한다:

I particularly like the section on automation because it states so clearly the basis on which automation should be judged. I’m working up what I think of as “reference designs” for Extreme programming acceptance tests. I will do my best to be sure they reflect nicely in the light of this book. Let me also say that I like very much the format of your book. It is what I had hoped the patterns community would routinely produce. The last time I saw a book I liked so much I asked to write the foreword. That was for David Thomas and Andrew Hunt’s “Pragmatic Programmer”. — Ward Cunningham

Chapter 1. The Role of the Tester

Lesson 8. You focus on failure, so your clients can focus on success.

Lesson 9. You will not find all the bugs

Lesson 10. Beware of testing “completely”

Lesson 11. You don’t assure quality by testing

Chapter 2. Thinking Like a Tester

Lesson 16. Testing is applied epistemology:

Epistemology is a branch of philosophy that helps you test better. … Epistemology is the study of how you know what you know. It’s the study of evidence and reasoning. It establishes the foundations of scientific practice. … Applied to software testing, epistemology asks questions like the following:

How do you know the software is good enough?

How would you know if it wasn’t good enough?

How do you know you’ve tested enough?

Lesson 17. Studying epistemology helps you test better:

Here are three very readable books to get you started: - ToolsOfCriticalThinking : Metathoughts for Psychology - ThinkingAndDeciding - TheCraftOfResearch|

이번엔 인지심리학(CognitivePsychology).

Lesson 18. Testing is grounded in CognitivePsychology

Some of its topics relating to testing include:

Reliability of your senses and memory.

Where beliefs come from.

How your beliefs affect your behavior.

Biases and short cuts you use to make decisions.

How you learn and share what you know.

How you think about complex things.

How you think under pressure.

How you recognize patterns.

How you sort ideas and things into categories.

How you notice differences between things.

Distortions in remembered events.

How you reconstruct partially remembered events (such as nonreproducible bugs).

Lesson 38. Use heuristics to quickly generate idea for tests.

Here are some examples of testing heuristics:

Test at the boundaries. Boundaries are more linkely to reveal ambiguities in the specification

Test every error message. Error-handling code tends to be weaker than mainstream functionality

Test configurations that are different from the programmer’s. The programmer will already be biased toward making sure his own configuration works.

Run tests that are annoying to set up. Easy-to-set-up tests are more likely to be performed, all other things being equal.

Avoid redundant tests. If a test is truly duplicating another test, what new value is it giving you?

Chapter 3. Testing Techniques

Chapter 4. Bug Advocacy

Lesson 65. Never use the bug-tracking system to monitor programmers performance.

Lesson 66. Never use the bug-tracking system to monitor testers performance.

Lesson 74. A failure is a symptom of an error, not the error itself.

The failure migh look minor, but the underlying problem might be much more serious. … Therefore, whenever you see an error that looks minor.

Do follow-up testing to find more serious symptoms, which will make for a more compelling report.

Do follow-up testing to show that the problem occurs under a wider range of circumstances and will be seen by lots of people.

Lesson 76. Always report nonreproducible errors; they may be time bombs.

Programmers have tools that you don’t have. If you report the symptoms clearly, the programmer can often trace through the code, asking how you could have gotten a certain message or what could possibly have happened when you were looking at a given dialog or clicking on a given control. …

Using PrintScreen, a screen recorder like Spector, or even video recording can help you prove the existence of UFOs (Unidentified Funny Objects) that some programmers would otherwise say could never have appeared.

Lesson 77. Nonreproducible bugs are reproducible.

Lesson 80. Ask before reporting bugs against prototypes or early private versions.

Lesson 83. The summary line is the most important line in the bug report.

Lesson 85. Report the problem clearly, but don’t try to solve it.

Your job is to report problems, not to identify root causes and not to push for specific solutions. Without studying the underlying code, you aren’t in a position to know the root cause of a failure. …

Some programmers are likely to reject reports whose “solutions” they know to be invalid, without considering the underlying problem. Don’t make it easy to reject your reports. …

Reports like these sometimes bring a lot of laughs to the programming group. You might not want to be the person at whose ignorant suggestions they’re laughing.|

Lesson 87. Make your reports readable, even to people who are exhausted and cranky.

A remarkable number of bugs are fixed (and deferred) in the last weeks of the project, by programmers who are working heavy overtime to get the product finished. On tough projects, the programmers are often sleep-deprived, under stress, and over-caffeinated. Write reports that these programmers can understand.

Lesson 88. Improve your reporting skills.

Compare closed bugs that were fixed and not fixed. Look for differences in the way they were reported. If you want yours fixed, report them like the ones that historically do get fixed.

Read programmers’ ( and others’ ) answers to bug reports. What makes them confused? Angry? Unreceptive? Appreciateve?

Lesson 91. Meet the programmers who will read your reports.

Lesson 93. When the programmer says it’s fixed, make sure it isn’t still broken

Lesson 94. Verify bug fixes promptly

Lesson 95. When fixes fail, talk with the programmer

Lesson 96. Bug reports should be closed by testers

Lesson 101. When you decide to fight, decide to win!

Talk with other stakeholders, such as Technical Support, Documentation, Sales, and so on. Find out whose budget will be mostaffected if the ubg is left in the product, how much it will cost them, or how bothered they are by it.

Chapter 5. Automating Testing

Lesson 109. Don’t estimate the value of a test in terms of how often you run it.

The value of a test comes from the information it provides.

Lesson 117. Automated regression tests die.

The biggest problem facing automated regression tests is decay and premature death.

Lesson 126. Don’t build test libraries simply to avoid repeating code.

Standard programming wisdom recommends avoiding repeating code by placing the repeated code in a function that is called from each place where the repeating code resided. In test automation, this approach often leads to trouble. The opposite approach, which leaves the repeated code in place, is called open coding. …

If you simply relocate the code you see repeated, you end up with a hodge-podge library.|

Lesson 127. Data-driven test automation makes it easy to run lots of test variants.

To test different inputs and input combinations with a common test procedure, use data-driven test automation. … After you’ve put together a data-driven test procedure, you can use it again and again to execute new tests. The technique is most powerful with product workflows that have lots of different data options. Use a more complex variant, keyword-driven automation, to support tests composed of varying sequences or several alternate paths.

Data-driven test automation supports nonprogramming testers. The automators create the data-driven test procedures, and the testers create the test data. In some cases, you might find it hard to automate the verification of the test results. Have your test procedure collect the results and present them in the context of the input data to simplify the manual results analysis.

Lesson 128. Keyword-driven test automation makes it easy for nonprogrammers to create tests.

Keyword-driven test automation builds on the data-driven technique. However, the tables contain directives (keywords), not just data. …

Using it avoids complex logic in your test scripts (Lesson 125). Because the tests are stored in spreadsheets, they’re often easy for nonprogrammers to create and review. Bucause the tasks to be used and tested are all that the tester specifies, he can concentrate on the tests, not the control language.

In contexts that require many nonprogrammers to create automated tests, we believe this is one of the better solutions. One drawback of this approach is that you can’t write a tes unless supported keywords already exist for the tasks you require. Defining and implementing the task functions can become a major undertaking.

Reviewer Hans Buwalda reports, “Keyword driven approaches, like my own action words, can provide a good basis. However, over the years I have learned that testing and test automation remain very challenging area and need the involvement of experienced specialists to do it right.” …

We also have reports of teams that found that this approach required too much overhead and was unsustainable.

Chapter 6. Documenting Testing

Chapter 7. Interacting with Programmers

Many testers have told us that programmers come back with excuses. “It will compromise the security of software.” “It will hurt performance.” These are occasionally valid concerns for test code. But usually, we believe, these are code phrases for “we don’t want to think about it.” —p150

Chapter 8. Managing the Testing Project

Lesson 162. There are always late changes.

Many traditional project management approaches are designed to limit and control change, but others embrace it (for example, see Weinberg 1992 (Quality software management Volume 1), Beck 1999 (Extreme programming explained), Beck et al. 2001 (Agile manifesto), and Krutchen 2000 (The rational unified process - An introduction) ). All project management approaches must nonetheless deal with change. …

Requirements are the result of ongoing struggles between what we want and what we can have. As the project progresses, the requirements change.

Lesson 177. “Project documents are interesting fictions: Useful, but never sufficient.”

Consider error handling. We’ve seen estimates that more than 80 percent of the code of a modern software project deals with error handling and less than 20 percent deals with primary flows. Yet even supposedly thorough specifications dedicate less than 20 percent of their space to error handling. That means that 80 percent of the code is designed by programmers as they write it.

Lesson 182. Great test planning makes late changes easy.

Here are a few suggestions:

Rather than developing a large suite of tests in advance of testing, develop tests as you need them. If later changes to the product obsolete these tests, at least they will have been useful for a while.

Don’t create enormous test documents that have high maintenance costs, such as detailed manual test scripts. Keep your paperwork as lean as psooible.

Don’t tie manual or automated tests to specifics of the user interface, other than tests specifically intended for the user interface. Even if you do end-to-end testing, which is necessarily through the user interface, don’t tie your test to the fnie details of the UI, because they will change.

Seriously consider using Extreme programming methods to develop automated tests. …

Lesson 192. Try testing in pairs.

Chapter 9. Managing the Testing Group

Chapter 10. Your Career in Software Testing

Lesson 249. Extend your career beyond Software testing

Lesson 250. Extend your career beyond your company

Lesson 269. Improve your writing skills

Chapter 11. Planning the Testing Strategy

Appendix. The Context-Driven Approach to Software Testing

The seven basic principles of the context driven school:

The value of any practice depends on its context.
There are good practices in context, but there are no best practices.
People, working together, are the most important part of any project’s context.
Projects unfold over time in ways that are often not predictable.
The product is a solution. If the problem isn’t solved, the product doens’t work.
Good software testing is a challenging intellectual process.
Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.